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Abstract 

We  present  two  new  classes  of  pattern  search  algorithms  for  unconstrained  min¬ 
imization;  the  rank  ordered  and  the  positive  basis  pattern  search  methods.  These 
algorithms  can  nearly  halve  the  worst  case  cost  of  an  iteration  compared  to  the  classi¬ 
cal  pattern  search  algorithms.  The  rank  ordered  pattern  search  methods  are  based  on 
a  heuristic  for  approximating  the  direction  of  steepest  descent,  while  the  positive  basis 
pattern  search  methods  are  motivated  by  a  generalization  of  the  geometry  characteris¬ 
tic  of  the  patterns  of  the  classical  methods.  We  describe  the  new  classes  of  algorithms 
and  present  the  attendant  global  convergence  analysis. 
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1.  Introduction.  In  this  paper  we  introduce  two  new  classes  of  pattern  search 
algorithms:  the  rank  ordered  and  positive  basis  pattern  search  methods  for  the  uncon¬ 
strained  minimization  problem 

minimize  fix). 

The  rank  ordered  and  positive  basis  pattern  search  methods  extend  the  analysis  devel¬ 
oped  in  [12].  These  new  classes  of  algorithms  can  almost  halve  the  worst  case  cost  of  an 
iteration  when  compared  with  the  classical  pattern  search  methods  considered  in  [12]. 
Moreover,  the  simple  heuristics  that  motivate  rank  ordered  and  positive  basis  pattern 
search  methods  are  intuitively  appealing  and  make  the  methods  straightforward  to 
describe. 

Pattern  search  methods  form  a  class  of  “steep  descent”  procedures  (a  term  we  will 
shortly  explain)  for  nonlinear  minimization.  Examples  include  [1],  [6],  and  [11].  While 
these  algorithms  have  no  explicit  recourse  to  a  Taylor  series  model  of  the  objective,  or 
any  information  about  directional  derivatives,  one  can  develop  a  global  convergence 
analysis  for  pattern  search  methods  with  results  similar  to  those  for  quasi-Newton 
methods. 

This  is  possible  because  pattern  search  methods  are  gradient-related,  so  that  when 
the  steps  become  small  enough,  they  are  are  guaranteed  to  capture  a  portion  of  the 
improvement  promised  by  the  steepest  descent  direction.  This  intuition  seems  to 
have  been  in  the  minds  of  the  early  developers  of  the  broader  class  of  direct  search 
methods;  for  instance,  Spendley,  Hext,  and  Himsworth  [9]  refer  to  their  algorithm  as  a 
method  of  “steep  ascent”  (they  were  considering  maximization)  to  indicate  its  kinship 
to  the  method  of  steepest  ascent.  The  convergence  analysis  for  pattern  search  methods 
confirms  that  this  is  particularly  apt  and  we  therefore  propose  the  term  “methods  of 
steep  descent”  to  describe  pattern  search  methods.- 

A  characterization  and  analysis  of  a  class  of  pattern  search  methods  is  presented 
in  [12].  Key  to  the  global  convergence  analysis  is  the  fact  that  these  methods  produce 
iterates  that  lie  on  a  rational  lattice.  To  ensure  global  convergence,  the  pattern  search 
methods  discussed  in  [12]  must  satisfy  a  condition  that  prevents  the  mesh  size  from 
being  reduced  if  one  of  the  steps  in  a  set  of  2n  core  steps  produces  decrease  in  the 
objective  value.  This  condition  allows  the  acceptance  of  any  step  that  lies  in  a  certain 
finite  subset  of  the  lattice  as  long  as  the  step  produces  simple  decrease  on  the  objective 
value  at  the  current  iterate,  and  plays  the  role  in  the  global  convergence  analysis  of 
pattern  search  methods  that  is  played  by  the  fraction  of  Cauchy  decrease  condition 
for  trust-regions  and  the  Armijo-Goldstein- Wolfe  conditions  for  line-searches. 

As  a  consequence,  one  can  construct  pattern  search  algorithms  that  require,  in  the 
best  case,  only  one  new  objective  value  per  iteration.  This  feature  is  used  to  advantage 
in  a  proposed  strategy  for  scientific  and  engineering  optimization  [4]  for  problems  in 
which  the  computational  cost  of  a  single  objective  evaluation  is  sufficiently  great  as 
to  merit  particular  care  in  choosing  steps  at  which  to  evaluate  the  objective.  On  the 
other  hand,  the  worst  case  occurs  when  it  is  necessary  to  reduce  the  mesh  size  to  make 
further  progress,  for  instance,  because  the  search  has  reached  a  neighborhood  either 
that  contains  a  local  minimizer  or  where  the  objective  is  highly  nonlinear.  For  the 
classical  pattern  search  algorithms  studied  in  [12],  as  many  as  2n  objective  evaluations 
may  be  necessary  in  this  case. 

The  rank  ordered  pattern  search  methods  and  positive  basis  pattern  search  meth¬ 
ods  that  we  introduce  here  improve  this  worst  case  bound  to  n  -I- 1  objective  values. 
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They  do  this  by  reducing  to  a  minimal  size  the  set  of  directions  that  we  must  consider 
to  be  assured  of  having  a  sufficiently  rich  set  of  search  directions  to  capture  a  suitably 
large  component  of  the  direction  of  steepest  descent. 

The  rank  ordered  pattern  search  methods  are  motivated  by  the  following  heuristic. 
Suppose  that  in  addition  to  the  objective  value  at  the  current  iterate  we  know  the 
objective  value  at  n  other  suitably  independent  points.  Then  the  direction  from  the 
point  with  the  highest  objective  value  to  the  point  with  the  lowest  objective  should 
be,  when  this  pattern  of  points  is  sufficiently  small,  a  crude  estimate  of  the  direction  of 
steepest  descent.  Unlike  a  finite-difference  approximation  to  the  direction  of  steepest 
descent,  this  approximation  ignores  the  distances  between  the  points  (and  thus  the 
relative  rate  of  change).  On  the  other  hand,  this  estimate  only  requires  that  we  identify 
the  best  and  worst  objective  values  among  a  set  of  n  +  1  values.  This  is  in  keeping 
with  a  distinctive  feature  of  pattern  search  methods:  namely,  that  they  only  require 
information  about  the  relative  rank  of  objective  values,  and  can  actually  be  used  in 
the  absence  of  any  numerical  objective  value,  as  might  be  the  case  when  the  algorithm 
is  driven  by  a  subjective  preference  on  the  part  of  the  user. 

The  analysis  in  [12]  does  not  make  use  of  rank-order  information,  though  it  is 
implicit  in  the  multidirectional  search  algorithm  considered  there.  As  it  happens,  the 
simple  heuristic  of  using  rank  ordering  and  the  notion  of  “steep  descent”,  described 
above,  to  suggest  search  directions  suffices  to  prove  global  convergence.  We  note  that 
this  heuristic  also  motivates  the  direct  search  method  of  Nelder  and  Mead,  which  is 
not  a  pattern  search  method  and  is  known  not  to  be  robust  [8],  and  the  aforemen¬ 
tioned  direct  search  method  of  Spendley,  Hext,  and  Himsworth  [9],  the  very  interesting 
analysis  of  which  we  will  discuss  elsewhere. 

The  heuristic  of  using  the  best  and  worst  objective  values  to  suggest  a  direction 
of  steep  descent  reduces  the  cardinality  of  the  core  set  of  steps  from  2n  to  n  +  1, 
but  may  introduce  an  inherently  sequential  component  into  the  algorithm.  It  may 
be  the  case  that  we  cannot  specify  the  final  search  direction  until  after  the  first  n 
points  have  been  evaluated  and  ranked.  The  positive  basis  pattern  search  methods 
have  the  attractive  feature  that  they  avoid  this  sequentiality  and  thus  are  well-suited 
for  parallel  implementation. 

It  was  in  the  course  of  re-examining  the  work  on  direct  search  methods  by  Yu 
Wen-ci  [13,  14]  that  we  realized  the  utility  of  the  theory  of  positive  linear  dependence 
developed  by  C.  Davis  in  [3]  for  generalizing  pattern  search  methods  in  a  useful  way. 
One  can  view  the  positive  basis  pattern  search  methods  as  the  natural  generalization 
of  the  algorithms  considered  in  [12]. 

We  say  “natural”  for  the  following  reason.  In  the  class  of  algorithms  studied  in 
[12],  the  2n  core  directions  played  the  technical  role  of  ensuring  that  we  would  search 
in  a  direction  that  made  a  positive  inner  product  with  the  direction  of  steepest  descent. 
The  notion  of  a  positive  basis  [3]  is  the  correct  way  to  generalize  this  latter  property, 
and  allows  us  to  reduce,  a  priori,  the  cardinality  of  the  core  set  of  steps  from  2n  to  as 
few  as  n-f  1.  Because  this  core  set  can  be  specified  in  advance,  we  avoid  the  sequential 
element  of  the  rank  ordered  pattern  search  methods. 

Notation.  We  denote  by  R,  Q,  Z,  and  N  the  sets  of  real,  rational,  integer,  and 
natural  numbers,  respectively. 

AU  norms  are  Eucfidean  vector  norms  or  the  associated  operator  norm.  We  define 
L{x)  =  {y  :  f(y)  <  We  will  denote  the  gradient  V f{x)  by  g{x)  and  the  gradient 
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at  iteration  k  g{xk)  hy  gk- 

By  abuse  of  notation,  if  A  is  a  matrix,  y  £  A  means  that  the  vector  y  is  a  column 
of  A.  We  will  also  use  c  and  C  to  denote  divers  constants  whose  identity  will  vary 
from  place  to  place  but  whose  nature  will  never  depend  on  the  iteration  k. 

2.  Pattern  Search  Methods.  Pattern  search  methods,  including  the  two  new 
classes  of  algorithms  under  discussion,  are  characterized  by  the  nature  of  the  generating 
matrices  and  the  exploratory  moves  algorithms,  these  features  are  discussed  more 
fuUy  in  [12]. 

To  define  a  pattern  we  need  two  components,  a  basis  matrix  and  a  generating 
matrix.  We  will  expand  the  class  of  pattern  search  algorithms  by  expanding  the  class 
of  admissible  generating  matrices. 

The  basis  matrix  can  be  any  nonsingular  matrix  B  €  The  generating 

matrix  is  a  matrix  Ck  €  where  pk>  n  +  1.  We  partition  the  generating  matrix 

into  components 

(1)  Ck  —  [  P/:  Lk  0  ]. 

We  require  that  Pfc  €  M,  where  M  is  a  finite  set  of  integral  matrices  with  fuU  row 
rank.  In  §2.1  and  §2.2  we  wiU  discuss  further  requirements  on  the  members  of  M;  we 
will  see  that  Tk  must  have  at  least  n+1  columns.  The  0  in  the  last  column  of  Ck  is 
a  single  column  of  zeros. 

A  pattern  Pk  is  then  defined  by  the  columns  of  the  matrix  Pk  =  BCk-  For 
convenience,  we  use  the  partition  of  the  generating  matrix  Ck  given  in  (1)  to  partition 
Pk  as  follows: 


Pk  =  BCk  =  [  BTk  BLk  0  ]. 

Given  Ak  G  R,  >  0,  we  define  a  trial  step  s\  to  be  any  vector  of  the  form 
s\  =  AkB4:^  where  c|  is  a  column  of  Ck-  Note  that  Bel  determines  the  direction  of 
the  step,  while  Ak  serves  as  a  step  length  parameter. 

At  iteration  k,  we  define  a  trial  point  as  any  point  of  the  form  xl  =  Xk  +  $1,  where 
is  the  current  iterate. 

Algorithm  1  states  the  pattern  search  method  for  unconstrained  minimization. 
To  define  a  particular  pattern  search  method,  we  must  specify  a  basis  matrix  B,  the 
generating  matrices  Ck,  the  exploratory  moves  to  be  used  to  produce  a  step  Sk,  and 
the  algorithms  for  updating  Ck  and  Afc. 

Algorithm  1.  The  pattern  search  method  for  unconstrained  minimization. 

Let  xo  e  R"  and  Aq  >  0  be  given. 

For  =  0, 1,  •  •  •, 

a)  Compute  f(xk). 

b)  Determine  a  step  Sk  using  an  unconstrained  exploratory  moves  algorithm. 

c)  If  f{xk  +  Sk)  <  f{xk),  then  Xk+i  =  Xk  +  -s*.  Otherwise  Xk+i  =  Xk- 

d)  Update  Ck  and  Ak- 

jf  j(^xk  +  Sk)  <  f{xk)  we  can  the  iteration  successful;  otherwise,  we  caH  the 
iteration  unsuccessful. 

We  have  the  following  Hypotheses  on  Unconstrained  Exploratory  Moves.  This  is 
the  same  as  in  [12].  The  specification  of  the  matrix  Ffe  in  (1)  is  different  from  that  in 
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[12],  however.  There,  Tk  was  required  to  be  a  n  x  2n  matrix  with  full  row  rank.  The 
Hypotheses  on  Unconstrained  Exploratory  Moves  require  that  before  an  iteration  is 
declared  unsuccessful,  we  must  examine  all  the  steps  in  the  core  pattern  determined 
by  A^BTk  for  a  lower  objective  value.  Thus,  for  the  algorithms  considered  in  [12],  in 
the  worst  case  one  would  need  to  compute  a  minimum  of  2n  objective  values  in  an 
iteration.  In  the  rank  ordered  and  positive  basis  pattern  search  methods,  however,  the 
size  of  Tk  can  be  as  smaU  as  n  x  (n  +  1),  which  has  the  effect  of  reducing  the  worst 
case  cost  of  an  iteration  to  n  +  1  objective  evaluations. 

Hypotheses  on  Unconstrained  Exploratory  Moves. 

1.  Sk  G  AkPk  =  ^kBCk  =  Aat  [jSFa;  BLk  0]. 

2.  If  min  {  f{xk  +  y)  \  y  ^  ^kBTk  }  <  f{xk),  then  f{xk  +  Sk)  <  f{xk)- 

Algorithm  2  specifies  the  rule  for  updating  Ak-  The  conditions  on  9  and  A  ensure 
that  0  <  ^  <  1  and  A^*  >  1  for  all  A^*  6  A.  Thus,  if  an  iteration  is  successful  it  may  be 
possible  to  increase  the  step  length  parameter  A^;,  but  Ak  is  not  allowed  to  decrease. 

Algorithm  2.  Updating  Ak- 

Let  r  6  Q,  r  >  1,  and  {wq.Wi,-  C  Z,  <  0,  and  wi  >  0,  i  =  1,  •  •  •,  T.  Let 

e  =  and  A^  G  A  = 

a)  If  }{xk  +  Sk)  >  f{xk)  then  Aa:+i  =  OAk- 

b)  If  f{xk  +  Sk)  <  f{xk)  then  Ajt+i  =  ><k^k^ 

2.1.  Rank  Ordered  Pattern  Search  Methods.  As  discussed  in  the  Introduc¬ 
tion,  one  can  use  the  best  and  worst  objective  values  in  a  portion  of  the  pattern  to 
suggest  an  intuitive  direction  of  steepest  descent  for  exploration.  This  heuristic,  which 
we  will  develop  formally  here,  ensures  that  when  Ak  is  sufficiently  small,  one  has  a 
trial  step  that  is  a  suitably  good  direction  of  descent. 

For  the  rank  ordered  pattern  search  methods,  the  generating  matrix  is  an  n  x  pk 
matrix,  >  n  +  1,  which  we  partition  as: 

Ck  ^  [  Sk  Rk  Lk  ^  [  Tk  Lk  ^  ]- 

We  require  Sk  to  be  a  nonsingular  element  of  The  notation  Sk  is  chosen  to 

suggest  simplex^  and  Rk  is  meant  to  suggest  reflection^  as  in  the  multidirectional 
search  algorithm  [11]. 

Let  Sk  =  [dl  dl'  —  d'^]  and  consider  the  simplex  with  vertices 

(2)  {xk,  Xk  +  AkBdl,  Xk  +  AkBdl,  •  •  • ,  Xfc  +  AkBd^}  =  vl,  Vfc,  •  •  • , 
where  the  vertices  are  ordered  (and  possibly  relabeled)  so  that 

(3)  fivk)  <  <  f{vk)- 

Note  that  we  only  need  to  identify  the  vertices  with  the  best  and  the  worst  objective 
values;  we  are  not  required  to  give  a  relative  ranking  of  the  remaining  n  —  I  vertices. 

Given  the  best  and  worst  objective  values  among  the  vertices  of  the  simplex  (2), 
we  can  then  say  how  Rk  is  chosen.  We  require  Rk  to  contain  a  column  of  the  form 

(4)  - <)  ’  pi  >0,  Pi  >  0. 

i=l 
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The  presence  of  a.  column  of  this  form  in  Rf^  means  that  the  core  pattern  will 

contain  a  trial  step  of  the  form 

(5)  <^k  =  ^Pk{'<’k-4)  ■ 

Since  the  column  defined  by  (4)  is  contained  in  =  [5"^  Rk],  there  are,  implicitly, 
further  restrictions  on  the  choice  of  i  =  l,---,n,  because  F/;  €  M,  and  M  is  a 
finite  set  of  integral  matrices  with  full  row  rank.  One  choice  would  be  to  set  p*  =  0  for 
i  =  1,  •  •  • ,  n  —  1  and  =  1.  An  example  can  be  seen  for  in  Fig.  4.  Another  choice 
would  be  to  set  pi  =  1/n  for  i  =  1,  •  •  •, n,  provided  that  the  Rk  that  results  is  integral. 
This  can  be  arranged  if  one  is  willing  to  scale  B  appropriately.  The  latter  choice  gives 
equal  weight  to  all  the  n  potential  descent  directions  (u|.  -  v^)  for  z  =  1,  •  •  • ,  n. 

The  requirement  that  >  0  ensures  that  at  least  one  of  the  search  directions 
defined  by  BRk  is  biased  towards  the  direction  from  the  worst  vertex  to  the  best. 
This  distinguished  direction  wiU,  when  Ak  is  small,  capture  enough  of  the  direction 
of  steepest  descent  to  allow  us  to  prove  global  convergence. 

Here  we  see  how  using  the  rank-order  information  and  the  steepest  edge  heuristic 
can  introduce  a  sequential  element  into  the  definition  of  Ck-  For  instance,  the  choice 
=  0  for  z  =  1,  •  •  •, ra  -  1  and  =  1  means  that  we  cannot  define  Rk  until  we  have 
identified  and  satisfying  (3). 

2.1.1.  An  example  of  a  rank  ordered  pattern  search  method.  In  Fig.  1 
we  give  an  example  of  a  rank  ordered  pattern  search  method.  This  algorithm  is 
a  sequential  variant  of  the  parallel  multidirectional  search  algorithm  of  Dennis  and 
Torczon  [5]. 

One  of  three  possible  steps  are  accepted  at  the  conclusion  of  each  iteration:  a 
reflection  step,  an  expansion  step,  or  a  shrink  step.  Examples  for  each  of  these  three 
steps  in  can  be  seen  in  Figs.  2,  3,  and  4,  respectively.  Note  that  since  we  know  the 
values  of  upon  entry  into  iteration  k,  this  fairly  conservative  variant 

of  a  rank  ordered  pattern  search  method  computes  exactly  n+\  objective  values  at 
each  iteration,  regardless  of  the  step  that  is  finally  accepted.  More  elaborate  variants, 
which  attempt  to  compute  a  single  objective  value  per  iteration,  are  possible.  The 
variant  we  present  here  has  the  advantage  of  being  both  simple  and  robust. 

2.2.  Positive  Basis  Pattern  Search  Methods.  The  positive  basis  pattern 
search  methods  will  be  described  in  terms  of  the  notion  of  positive  linear  dependence 
developed  in  [3].  Positive  linear  dependence  captures  the  essential  technical  role  played 
by  the  core  pattern  in  [12]. 

The  positive  basis  pattern  search  methods  are  also  motivated  by  the  requirements 
of  parallel  computing.  As  we  have  seen,  the  rank  ordered  pattern  search  methods  may 
introduce  a  sequential  element  into  the  computation  since  we  must  satisfy  (4).  As  we 
have  seen,  this  may  require  the  identification  of  the  worst  vertex  before  we  can  append 
the  remaining  search  direction  in  the  pattern.  As  a  consequence,  Amdahl’s  Law  says 
that  we  can  at  most  halve  the  execution  time  of  a  single  iteration  via  computational 
parallelism,  regardless  of  the  number  of  processing  units  available.  The  positive  basis 
pattern  search  methods  avoid  this  sequentiality  by  imposing,  a  geometric  condition 
on  F*.  Thus  the  positive  basis  pattern  search  methods  are  ideally  suited  for  parallel 
implementation  since  the  algebraic  conditions  on  Ffc  can  be  satisfied  a  priori  and 
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Start  with  an  initial  simplex  with  vertices  {vq,?;q,  • 
and  evaluate  /(t'o)?  J  =  0,  •  •  •,  n. 
for  A:  =  0, 1,  •  •  • 

Reorder  the  vertices  of  the  simplex  so  that 


Check  the  stopping  criteria. 


rk 


2.” 


evaluate  f{rk) 
if  (/(rfc)  <  f{vl))  then 
e,^Zvl-2vl 
evaluate  f{ek) 
if  (/(e*)  <  f{rk))  then 

for  j  =  1, 

J 


•,n  —  1 


'^k+l 


evaluate 

end  for 


else 


'^k+i 


n 


for  j  =  1,  •  •  • ,  n  -  1 

4+1  ^^4- vi 

evaluate 
end  for 
end  if 


else 


for;  =  1, 


^k+i 


••,n 

1  ,,o  I  1  J 

2  '  2 


end 
end  if 


evaluate 


end 


/*  reflection  step  */ 


/*  expansion  step  */ 


/*  accept  expansion  */ 
/*  expand  simplex  */ 


/*  accept  reflection  */ 
/*  reflect  simplex  */ 


/*  shrink  simplex  */ 


Fig.  1.  A  straightforward  rank  ordered  pattern  search  method. 


allow  all  the  necessary  objective  evaluations  for  a  single  iteration  to  be  independently 
computed  in  parallel. 

2.2.1,  Positive  linear  dependence.  We  present  here  the  ideas  we  will  need 
from  the  theory  of  positive  linear  dependence  [3].  The  positive  span  of  a  set  of  vectors 
{ai,  •  •  • ,  a^}  is  the  cone 

{  a  €  1  G  =  CiUi  H - h  CrOr^  Ci  >  0  for  all  z  }  . 

The  set  {ui,  •  •  *,^7.}  is  called  positively  dependent  if  one  of  the  ttj’s  is  a  nonnegative 
combination  of  the  others;  otherwise  the  set  is  positively  independent  A  positive  basis 
is  a  positively  independent  set  whose  positive  span  is  R^.  The  following  theorem  from 
[3]  indicates  that  a  positive  spanning  set  contains  at  least  n  +  1  vectors. 

Theorem  2.1.  Suppose  {ai,  •  •  -  positively  spans  R^.  Then  {02,  •  •  *,0^7-}  lin¬ 
early  spans  R^. 
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Fig.  2.  A  reflection  step  of  the  straightforward  rank  ordered  pattern  search  method,  given  the  simplex 
with  vertices  {v\,v\,v\}. 

The  following  characterizations  of  positive  spanning  sets  can  be  found  in  [3]  as  well. 

Theorem  2.2.  Suppose  {ai,  •  •  • ,  ^  0,  linearly  spans  R”.  Then  the  follow¬ 

ing  are  equivalent: 

1.  {ci,  •  •  • ,  Or}  positively  spans  R". 

2.  For  every  b  ^  0,  there  exists  an  i  for  which  b^ai  >  0. 

3.  For  every  i  =  -ai  is  in  the  convex  cone  positively  spanned  by  the 

remaining  ai. 

A  positive  basis  that  contains  n  +  1  elements  is  called  minimal.  One  can  also  show 
[3]  that  a  positive  basis  can  have  no  more  than  2n  elements;  such  a  basis  is  called 
maximal.  A  maximal  positive  basis  has  a  very  special  structure:  it  must  consist  of  a 
linear  basis  for  R”  and  the  negatives  of  those  basis  vectors.  On  the  other  hand,  it 
is  easy  to  see  that  such  a  collection  of  2n  vectors  is  a  positive  basis,  as  the  next  two 
propositions  show. 

Proposition  2.3.  Suppose  {ai,'--,ar}  is  a  positive  basis  for  R”,  and  B  is  a 
nonsingular  n  x  n  matrix.  Then  the  set  {Rci,  •  •  •,  Bar}  is  also  a  positive  basis  for  R”. 

Proof.  Because  B  is  nonsingular,  the  set  {Bai,  •  •  •,  Bar}  linearly  spans  R".  More¬ 
over,  because  is  a  positive  basis,  by  Theorem  2.2,  part  (2),  given  any 

b  E  R”,  b  ^  0,  there  exists  an  i  for  which 

(B^bfai  =  b^Bai  >  0. 

But  this  means  that  {Bai,-  ■  ■ ,  Bar}  is  a  positive  basis  for  R".  □ 

Proposition  2.4.  If  B  is  a  nonsingular  n  x  n  matrix,  then  the  columns  of  the 
matrix  [B  —  B]  form  a  positive  basis. 

Proof  By  part  (2)  of  Theorem  2.2,  we  know  that  the  matrix  [/  -  /]  is  a  positive 
basis.  The  result  then  follows  from  Proposition  2.3.  □ 


Fig.  3.  An  expansion  step  of  the  straightforward  rank  ordered  pattern  search  method,  given  the  simplex 
with  vertices  {v\,v\,v\}. 


2.2.2.  Description  of  the  positive  basis  pattern  search  algorithm.  In  a 

positive  basis  pattern  search  method,  we  require  the  generating  matrix  Ck  to  be  any 
nxpk  matrix,  pk  >  ra  +  1,  of  the  form  found  in  (1),  but  now  we  require  Ffc  to  be  a 
positive  basis  for  R”.  Proposition  2.-3  then  says  that  at  every  iteration  k,  will 

also  be  a  positive  basis  for  R”. 

2.2.3.  Examples  of  positive  basis  pattern  search  methods.  The  classical 
pattern  search  methods  considered  in  [12] — coordinate  search  with  fixed  step  length, 
Hooke  and  Jeeves,  and  Evolutionary  Operation  using  factorial  designs — are  all  positive 
basis  pattern  search  methods  as  we  have  defined  them.  The  generating  matrix  in  [12] 
has  the  form 


Ck  =  [  Mk  -Mk  Lk  0  ] 

for  some  n  x  n  nonsingular  matrix  Mk-  In  light  of  the  discussion  in  §2.2,  the  nature 
of  =  [Mk  —  Mk]  as  a  maximal  positive  basis  is  now  revealed. 

We  are  not  aware  of  any  classical  pattern  search  method  that  uses  a  positive  basis 
that  is  not  maximal.  Coordinate  search  with  fixed  step  length,  Hooke  and  Jeeves,  and 
Evolutionary  Operation  using  factorial  designs  all  use  positive  bases  with  2n  elements. 

But  it  is  not  difficult  to  use  the  general  notion  of  a  positive  bcisis  to  invent  new 
pattern  search  methods  that  are  proper  extensions  of  the  class  of  algorithms  studied 
in  [12].  Such  methods  have  a  practical  appeal  because  we  can  reduce  the  number  of 
objective  evaluations  in  the  worst  case  from  2n  to  as  few  as  n  +  1. 

Here  are  two  possible  choices  for  Tk  that  lead  to  minimal  positive  basis  pattern 
search  methods.  Let  e  =  (1, 1,  •  •  • ,  1)^,  and  consider 

(6)  Tk  =  [nl  -  e] 
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Fig.  4.  A  shrink  step  of  the  straightforward  rank  ordered  pattern  search  method,  given  the  simplex 
with  vertices  {v%,v\,v\}. 


and 

(7)  Tk  =  [I  -  e]. 

It  is  not  difficult  to  see  from  Theorem  2.2,  part  (2),  that  the  columns  of  each  matrix 
form  a  positive  spanning  set,  and  since  there  are  n  +  1  columns  in  each  case  these 
positive  spanning  sets  are  minimal  positive  bases. 

Consider  first  the  choice  (6).  If  the  basis  matrix  is  given  by  5  =  i[6i  •  •■bn],  then 

1  " 

BTk  =  [bi---bn  bn+i]  where  6„+i  =  -  -  X!  • 

^  t=i 

Thus,  the  trial  steps  in  this  pattern  are  the  vectors  together  with  the 

negative  of  their  average.  In  Fig.  5  we  illustrate  an  example  of  such  a  minimal  positive 
basis  for  R^. 


Fig.  5.  A  minimal  positive  basis  for  with  basis  vectors  bi  =  (v\,  -  Xk)  for  i  =  1,  ■  ■  ■  ,3. 
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For  the  choice  (7),  suppose  J9  =  [6i  •  ••bn]-  Then 

n 

BTk  =  [bi-- -bn  6„+i]  where  =  -  ^  6,-. 

i=l 

This  pattern  has  an  interpretation  in  terms  of  a  simplex.  We  claim  that  these  vectors 
are  the  vectors  from  the  centroid  of  a  simplex  to  the  vertices  of  the  simplex.  If 
7  *  *  *  9  ^n+1  Sire  the  vertices  of  a  simplex  and 


V  = 


1 

n  +  1 


i=o 


is  the  centroid,  we  wish  to  know  whether  we  can  find  Vi  for  vrhich  b{  =  Vi  ~  v.  This  is 
the  same  as  seeking  a  solution  of  the  system  of  equations 


where 

V  =  [ui-’-Vn+i] 

B  =  [hi  •  • 


Let 


^  =  7- 


71+  I 


-ee 


then  we  are  asking  whether  we  can  solve  VA  =  B.  We  have  the  following  alternative: 
either  VA  =  B  has  a  solution,  or  there  exists  y  such  that  Ay  =  0,  By  7^  0.  However,  A 
and  B  have  the  same  nuUspace:  namely,  that  spanned  by  the  vector  e.  Thus,  it  must 
be  the  case  that  we  can  solve  VA  =  B^  which  means  we  can  interpret  the  pattern  as 
the  vectors  from  the  centroid  of  a  simplex  to  the  vertices  of  the  simplex. 

Having  noted  two  possible  positive  bases  that  satisfy  our  requirements  for  a  pos¬ 
itive  basis  pattern  search  method,  in  Fig.  6  we  give  an  example  of  a  positive  basis 
pattern  search  method.  Note  that  this  uses  an  exploratory  moves  strategy  that  is 
identical  to  that  used  for  the  Evolutionary  Operation  algorithm  (see  either  [2],  [10], 
or  [12]).  but  instead  of  using  a  two-level  factorial  design,  which  requires  2^  objective 
values  per  iteration,  the  positive  basis  allows  us  to  implement  a  design  with  as  few  as 
71  +  1  objective  values  per  iteration. 

2.2.4.  The  relationship  between  positive  basis  and  rank  ordered  pat¬ 
tern  search  methods.  There  is  an  overlap  between  the  class  of  positive  basis  pattern 
search  methods  and  rank  ordered  pattern  search  methods,  as  we  shall  soon  demon¬ 
strate.  However,  it  is  instructive  to  see  that  the  two  classes  are  not  equivalent. 

To  see  that  we  cannot  necessarily  cast  a  rank  ordered  pattern  search  method  as 
a  positive  basis  pattern  search  method,  we  return  to  the  example  given  in  §2.1.1, 
where  ~  ^  legitimate  reflection  step  for  a  rank 

ordered  pattern  search  method  since  it  satisfies  (4)  and  (5).  Fig.  7  shows  that  the 
set  {{vl  —  u®),  {vl  -  {vk  —  does  not  form  a  positive  basis  for  because  we 
cannot  find  a  member  of  the  set  for  which  the  inner  product  with  the  vector  {b  — 
is  strictly  positive.  Thus  we  violate  part  (2)  of  Theorem  2.2. 
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Start  with  an  initial  point  a;o,  a  positive  basis  BV  =  [B'i^  •  ••-67^],  and  Aq  >  0. 
Evaluate  f(xo). 
for  =  0, 1,  -  •  - 

Check  the  stopping  criteria. 

for  i  =  1,-  •  ■,p  /*  this  loop  can  be  parallelized  */ 

si  =  '^kBf 

evaluate  /(xfc  +  s^) 

end  for 

Sk  =  argmin{/(.'Cfc  +  -5^)}  /*  the  sole  synchronization  point  */ 

if  f{xk  +  Sk)  <  f{xk)  then 

Xk+\  =  Xk  +  Sk 

Afe+i  =  Afc 

else 

Xk+l  =  Xk 

Afc+a  =  |Afc 

end  if 

end 

Fig.  6.  A  straightforward  positive  basis  pattern  search  method. 


To  see  that  it  is  not  always  possible  to  cast  a  positive  basis  pattern  search  method 
as  a  rank  ordered  pattern  search  method  consider  the  example  shown  in  Fig.  8.  Here 
the  set  of  vectors  {(r^  -  ng),  (nj  -  (v|  -  n^)}  does  form  a  positive  basis  for  R^,  but 

the  vector  (rk-Vk)  does  not  constitute  an  acceptable  reflection  step  for  a  rank  ordered 
pattern  search  method.  The  difficulty  in  this  instance  is  that  (ffc  -  v^)  is  orthogonal 
to  {vl  -  vl),  thus  violating  conditions  (4)  and  (5)  which  our  analysis  imposes  upon 
rank  ordered  pattern  search  methods. 

As  can  be  seen  in  Fig.  9,  the  multidirectional  search  (MDS)  algorithm  lies  in  the 
intersection  of  the  two  approaches.  The  set  of  vectors  {(nj  -  v°),  •••,(«?-  (Vk  - 

positive  basis  for  R”.  We  can  implement  MDS  as  a  parallel  algorithm  by  simultane¬ 
ously  computing  the  objective  values  at  all  2n  points  defined  by  the  simplex  and  its 
reflection.  In  doing  so,  a  priori  we  treat  all  the  vertices  of  the  original  simplex,  ex¬ 
cept  the  best  vertex  as  possibly  being  the  worst  and  thus  remove  the  sequential 
bottleneck  or  any  need  to  coordinate  such  information  across  processes.  This  is  equiv¬ 
alent  to  implementing  a  positive  basis  pattern  search  method  with  a  maximal  positive 
basis.  However,  we  can  certainly  also  implement  MDS  as  a  sequential  algorithm,  as 
demonstrated  in  §2.1.1,  and  take  full  advantage  of  rank  order  information  in  an  effort 
to  reduce  the  number  of  function  evaluations  per  iteration.  We  do  so,  however,  at  the 
expense  of  a  sequential  element  that  limits  the  effective  use  of  parallel  computing. 

We  close  by  noting  that  the  convergence  analysis  for  pattern  search  methods  is 
flexible  enough  to  accommodate  a  myriad  of  other  combinations  that  may  perhaps  be 
more  appropriate  for  a  given  application.  We  have  illustrated  only  a  few  of  the  many 
options. 

3.  Convergence  analysis.  The  convergence  results  for  positive  bcisis  and  rank 
ordered  pattern  search  methods  are  like  those  in  [12]: 
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Fig.  7.  A  demonstration  that  a  rank  ordered  pattern  search  method  may  not  necessarily  he  a  positive 
basis  pattern  search  method 

Theorem  3.1.  Suppose  L(xo)  is  compact  and  suppose  f  is  continuously  differ¬ 
entiable  on  an  open  neighborhood  0,  of  L{xo)-  Let  {xk}  be  the  sequence  of  iterates 
produced  by  either  a  rank  ordered  or  a  positive  basis  pattern  search  method  for  uncon¬ 
strained  minimization  (Algorithm  1). 

Then 


Uminf  II  5(xjt)  II  =  0. 

/c— >-+oo 


We  will  henceforth  assume  that  /  is  continuously  differentiable  on  an  open  set 
9,  containing  the  compact  set  L{xq).  However,  for  Theorem  3.1  (and  Theorem  3.2 
to  foUow)  we  really  only  need  /  to  be  continuously  differentiable  on  L{xq).  Under 
that  assumption,  one  would  need  first  to  derive  estimates  that  show  that  if  xq  is  not  a 
stationary  point  then  a  pattern  search  method  eventually  wiU  move  into  the  interior  of 
L{xo)-  One  would  then  derive  a  similar  set  of  estimates  and  show  that  these  estimates 
are  uniform  on  X(xi),  in  the  interior  of  L{xo).  The  assumption  that  /  is  continuously 
differentiable  on  a  set  slightly  larger  than  i(xo)  makes  the  proof  shorter  and  clearer 
at  little  expense  of  generality. 

As  in  [12],  we  can  also  obtain  a  stronger  result.  To  do  so,  we  must  ultimately 
stop  allowing  Afc  to  become  larger,  and  we  must  require  that  the  algorithm  examine 
all  the  points  of  the  core  pattern,  according  to  the  Strong  Hypotheses  on  Exploratory 
Moves. 

With  these  restrictions,  we  obtain  the  following  result: 

Theorem  3.2.  Suppose  L{xo)  is  compact  and  suppose  f  is  continuously  differ¬ 
entiable  on  an  open  neighborhood  9  of  L{xq).  In  addition,  assume  that  the  columns 
of  the  generating  matrices  are  uniformly  bounded  in  norm,  that  in  the  update  of  Ak, 
we  have  Afc  =  1  for  all  k  after  some  iteration,  and  that  the  pattern  search  method 
for  unconstrained  minimization  (Algorithm  1)  enforces  the  Strong  Hypotheses  on  Ex¬ 
ploratory  Moves.  Finally,  let  {xk}  be  the  sequence  of  iterates  produced  by  either  a  rank 
ordered  or  a  positive  basis  pattern  search  method  for  unconstrained  minimization. 
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Fig.  8.  A  demonstration  that  a  positive  basis  pattern  search  method  may  not  necessarily  he  a  rank 
ordered  pattern  search  method 


Fig.  9.  A  reflection  step  for  a  parallel  implementation  of  the  multidirectional  search  algorithm. 


Then 

Um  1U(XA:)  II  =  0. 

»-+oo 


By  restricting  the  update  of  Ak  to  allow  only  Afc  =  1  for  all  A:  after  some  iteration, 
we  are  assured  that  lim;;_,+oo  Ak  =  0.  This  is  a  corollary  of  Theorem  3.18. 

The  outline  of  the  proof  for  Theorem  3.1  follows  that  of  [12]  as  developed  in  [7]: 

1.  First  we  show  that  given  rf  >  0,  there  exists  ^  >  0  (independent  of  k)  such 
that  if  II  g{xk)  \\  >  V  and  <  S,  then  a  pattern  search  algorithm  will  find 
an  acceptable  step  without  further  decrease  of  Ak- 

2.  We  then  show  that  if  liminffc_^+oo  ||  gi^^k)  ||  #  0,  then  there  exists  a  nonzero 
lower  bound  on  Afc. 

3.  Using  purely  algebraic  properties  of  iterates  produced  by  pattern  search  meth¬ 
ods,  we  show  that  we  must  necessarily  have  liminffc_,+oo  Ak  =  0. 
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strong  Hypotheses  on  Exploratory  Moves. 

1.  Sk  €  AkPk  =  AkBCk  =  Ak  BLk  0]. 

2.  If  min{/(xfc  +  y),  y  €  A^Srit}  <  f{xk),  then 
f(xk  +  sk)  <  mm{f{xk  +  y),  ye  AkBTk}- 

4.  We  then  conclude  that  we  must  have  liminffc-.+co  ||  di^k)  ll  =  0. 

The  analysis  of  positive  basis  pattern  search  methods  and  that  of  rank  ordered  pattern 
search  methods  differ  only  in  the  first  step.  After  that  the  two  lines  of  analysis  converge 
and  are  more  or  less  identical  to  similar  steps  in  [12]. 

The  sections  that  follow  correspond  to  the  outline  given  above.  The  new  results 
can  be  found  in  §3.1,  where  we  develop  the  case  first  for  the  positive  basis  pattern 
search  methods  and  then  for  the  rank  ordered  pattern  search  methods.  The  remaining 
sections  complete  the  analysis,  relying  largely  on  results  developed  in  [7]  and  [12]. 

3.1.  Existence  of  a  direction  of  descent.  The  following  proposition  is  the 
justification  for  calling  pattern  search  algorithms  “methods  of  steep  descent.” 

Proposition  3.3.  There  exists  c  >  0  such  that  given  any  rj  >  0,  we  can  find 
1/  >  0  such  that  if  ||  g{xk)  ||  >  ?/  o,nd  Ak  <  v,  there  is  a  step  s\  G  AkBTk  for  which 


9k  4  >c\\9k 


Sk 


The  proof  of  this  result  differs  for  positive  basis  pattern  search  methods  and  rank 
ordered  pattern  search  methods  and  thus  will  be  developed  independently  for  each. 

In  both  cases  recourse  wiU  be  had  to  the  following  result,  the  proof  of  which  can 
be  extracted  from  §6.2  of  [12]. 

Proposition  3.4.  Suppose  that  {ai,---, an}  linearly  spans  BP.  Then  given  any 
X  G  R",  we  can  find  an  ai  for  which 


T 

X  ai 


> 


1 


K{A)y/n 


Qi 


where  /v(A)  is  the  condition  number  of  the  matrix  A  =  [a\-  ■  -a^]. 

We  will  also  need  the  following  proposition,  which  says  that  we  can  uniformly 
bound  the  first-order  Taylor  series  remainder. 

Proposition  3.5.  Given  s  >  0,  we  can  find  S  >  0  such,  that  if  x  £  L{xq)  and 
i|  ?/  -  a;  II  <  then 


f{y)-  f{x)-9{^fiy-^) 


<£||  11- 


Proof.  Let  (5i  =  imin(l,dist(5T(a;o),^fi)).  If  a:  G  L{xo)  then  the  ball 
B{x,6i)  =  {  y  I  II  y-.T  II  <  (5i  } 


is  contained  in  Q,.  Let  K  be  the  closure  of  Uj.g£,(j.(,)5(a;,  ^i);  note  the  K  is  compact  by 
construction. 

Now,  if  a;  G  L{xo)  and  y  G  B{x,6i)  we  may  apply  the  mean- value  theorem  to 
obtain 


f{y)-  f{x)  =  9{zf{y- 
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for  some  2:  on  the  line  segment  connecting  x  and  y.  Then 

f{y)-  =  \{9{z)-  9{^)f  {y-^) 

<  II  II II 2/ -  ®  II 

The  uniform  continuity  of  g  on  /i  allows  us  to  find  ^2  ^  0  such  that  \f  x^y  6  A  and 
II  y  -  X  II  <  ^2  then 

II  9{z)-g{x)  II  <  £. 

Thus,  if  we  choose  b  =  min(^i,^2)?  then  if  x  G  i(a;o)  ||  ?/  ~  2;  ||  <  ^,  we  have 
f{y)  -  /(x)  -  g{xf{y  -  a;)  I  '<  £  li  J/  -  a;  II . 

0 

3.1.1.  The  case  of  positive  basis  pattern  search  methods.  For  positive 
basis  pattern  search  methods,  Proposition  3.3  follows  from  purely  geometric  properties 
of  the  pattern  without  reference  to  the  objective  and  the  requirement  that  be 
sufficiently  small. 

We  begin  by  generalizing  Proposition  3.4  to  positive  spanning  sets. 

Proposition  3.6.  Given  any  set  {ai ,  •  •  • ,  a,r}  that  positively  spans  R",  Oi  ^  0  for 
i  =  1,  •  •  • ,  r,  there  exists  c  >  0  such  that  for  all  x  G  R”,  we  can  find  an  ai  for  which 

x^ai  >  c  II  X  II  II  ai  II . 

Proof  We  need  consider  only  the  case  x  ^  0.  According  to  Theorem  2.1,  we  can 
find  a  basis  for  R"  among  the  vectors  {ai,-  •  -  ,0^}.  Thus,  possibly  upon  reordering, 
we  may  assume  that  the  matrix 

A  =  [fli  •  •  •  On] 


is  invertible.  Then  by  Proposition  3.4,  given  any  x  G  R”  we  can  find  a  column  a,  in 
A  for  which 


T 

X  ai  > 


X  ai  . 


There  are  two  possibilities  to  consider  in  (8).  The  first  is  that 


there  is  nothing  more  to  be  done  in  this  case. 
The  other  possibility  is  that 


— x^a,  > 


X  ai  . 


In  this  case,  we  first  appeal  to  part  (3)  of  Theorem  2.2  to  choose  a  set  of  scalars 
g-jk  >  O5  depending  only  on  {ai,-  •  -la-r},  such  that  for  any  j  we  can  express  —aj  as 

r 
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This  representation  for  —ai  leads  to 


^  lHhX^ak  > 


n{A)^/n 


X  tti 


k=l,k:^i 

It  must  then  be  the  case  that  for  some  index  i  we  have 


mex^ai  > 


1 


Since  a:  7^  0  we  know  that  it  must  be  the  case  that  nu  >  0,  so  we  arrive  at 

x'^ac  >  — - ^  (}\  r  II  ^  II  II  II  • 

r  —  1  K{A)y/n 


If  we  let 


fi*  =  max{  Hjk  I  l<j,k<r,j:/:k} 


and 


we  obtain 

(10) 


a,  =  min 


II  II 
II  cik  II 


1  <  j,k  <  r 


1 


X^Ui  > 


G*  _J. _ 1 

fi*  r  —  1  K{A)y/n 


ae  II . 


Combining  (9)  and  (10)  then  yields  the  proposition.  □ 

Proposition  3.3  in  the  case  of  positive  basis  pattern  search  methods  now  follows 
from  the  following  stronger  result. 

Proposition  3.7.  For  a  positive  basis  pattern  search  method  there  exists  c  >  0 
such  that  for  any  k,  there  is  a  step  s\  €  for  which 


-9k  4  >c\\9k  II 

Proof  At  each  iteration  k,  BTk  is  a  positive  basis.  Thus,  by  Proposition  3.6, 
there  exists  C{BTk)  >  0  for  which  we  can  find  v  G  BTk  such  that 

-9lv>C{BTk)\\gk  II  II  HI- 

However,  Tk  is  a  member  of  the  finite  set  of  matrices  M,  so  there  are  only  finitely 
many  possibilities  for  BTk-  Taking 


c  =  min  C(BTk)  >  0 

r;t€M  ^ 

and  multiplying  v  by  yields  the  proposition. 
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3.1.2.  The  case  of  rank  ordered  pattern  search  methods.  For  rank  ordered 
pattern  search  methods,  we  are  only  assured  of  a  suitable  descent  direction  when  A* 
is  sufficiently  small.  When  is  sufficiently  small,  the  relative  ranking  of  the  extreme 
objective  values  (best  and  worst  or  lowest  and  highest)  determines  a  direction  that  is 
suitably  close  to  the  direction  of  steepest  descent. 

Because  the  matrices  Tk  come  from  a  finite  set  of  matrices,  the  relative  sizes  of 
steps  in  the  core  pattern  remain  bounded  and  are  related  to  Ajt: 

Proposition  3.8.  There  exist  r,Ji>0  such  that  for  all  k,  if  s],  €  Afc.Prfc,  then 

rAk  <  II  I  <  RAk- 

Consequently,  given  e  >  0,  there  exists  6  >  0,  independent  of  k,  such  that  if 
Ak  <  S,  then  ||  ||  <  e  for  all  4  £  AkBTk. 

Proof  The  restrictions  on  Tk  for  both  the  rank  ordered  and  the  positive  basis 
pattern  search  methods  ensure  that  none  of  the  columns  of  Pfc  can  be  zero.  Since  B  is 
invertible,  BTk  can  never  have  a  zero  column.  Because  Fyi;  G  M  and  M  is  a  finite  set 
of  matrices  it  follows  that  there  exist  a  nonzero  lower  bound  and  a  finite  upper  bound 
for  the  norm  of  all  the  columns  of  elements  of  BM.  The  proposition  follows.  □ 

If  we  let  7*  =  r/R  and  7*  =  R/r,  then  Proposition  3.8  means  that  for  any 
4?4  ^  ^hBTk,  we  have 

(11)  7*  1 4 II  ^  II 4 1  <  7*  1 4 1  • 

Proposition  3.9  translates  the  rank  ordering  of  the  vertices  into  a  statement  about 
directions  of  descent.  It  says  that  when  Ajt  is  small  enough,  the  edges  from  the 
vertices  4’  ‘  4  vertex  are  either  descent  directions  or  are  not  very 

steep  ascent  directions. 

Proposition  3.9.  Given  e  >  0  and  g  >  0,  there  exists  1/  >  0  such  that  if 

II  5(4)  II  >  5  Ihen 

5(4)^(4  -  4)  <  £  II  5(4)  II II 4  -  4 II  i  = 

Proof  We  have  /(4)  -  /(4)  <  0  for  all  i  =  1,  •  •  •, n.  Applying  Propositions  3.-5 
and  3.8  we  can  find  1/  >  0  such  that  if  Ak  <  v,  then 

5(4)’'(4  -  4)  -  ^5 1 4  “  4 1  <  /(4)  -  /(4)  <  0- 

The  result  follows.  0 

The  next  step  is  to  show  that  when  Ajt  is  sufficiently  small,  the  direction  from  the 
worst  vertex  to  the  best  vertex  is  a  direction  of  steep  descent  from  4- 

Proposition  3.10.  There  exists  c  >  0  such  that  given  any  7  >  0,  we  can  find 
1/  >  0  such  that  if  ||  ^(4)  ||  >  5  <  v,  then 

-5(4)^(4  -  4)  >  c  II  5(4)  1 1 4  -  4 1  • 

Proof.  Let 

K*  =  max  K(BSk)', 
s.cr^eM 

where  by  Sk  C  Tk  we  refer  to  the  partition  Pjt  =  [5jt  .Rjt].  By  Proposition  3.4,  there  is 
some  index  i,  1  <  £  <  n,  for  which 

5(4)^(4  -  4)  I  >  II  5(4)  1 1 4  -  4  I  • 

17 


If  Afc  is  sufficiently  small,  we  can  divine  the  sign  of  the  inner  product:  by  Proposi¬ 
tion  3.9,  we  can  find  i^i  >  0  such  that  if  Ak  <  r'l,  then 


giving  -  4)  < 


1  1 
2  K*^/n 


9{vt) 


4-4 


This  means  that  we  must  actually  have 


(12) 


gi4fi4  -  4)  > 


K*y/n 


g(vt) 


4-4 


This  shows  the  existence  of  a  good  descent  direction.  Now  we  relate  (12)  to  the 
distinguished  direction  -  v^.  Because  f(vl)  <  f{v^)  for  aU  i, 


f{vt)  -  m)  <  f{vi)  -  f{vi). 


Using  this  inequality  and  Propositions  3..5  and  3.8,  given  any  £  >  0  we  can  find  1/2  >  0 
such  that  if  Ak  <  1^2  then 


<  fi4)  -  fi4)  <  g{4V{4  -4)  +  ^ 


4-4 


or 


(13)  -  -  4)  >  -9i4f  i4  -4)-^  (1 4  - ' 

Now  choose 

1  1 


-i- 


)• 


e  = 


4  K*y/n 


7?min(l,7*), 


and  U2  accordingly,  where  7*  is  as  in  (11).  Let  v  =  min(i/i,r/2).  Then,  if  Ak  <  v,  (12) 
holds.  Meanwhile, 


(14) 


£  4-4 


1  1 

< - y=T] 

4  K*^/n 

e\\vl-vl\\  < 


'^k  ^k\\  -  4 

Then  (12),  (13),  and  (14)  yield 


4-4 


< 


4  K*^/n 


4-4 


9{4)  i4-4)  >  o 


1  1 


2  K*y/n 


9{4)  4  -  4 


Applying  (11)  again  yields 


9{4f{4  -  4)  >  1* 


1  1 


2 


9{vt) 


4-4 


which  is  the  desired  estimate.  □ 

We  can  now  attend  to  the  reflection  step  ak  from  (.5),  which  may  comprise  all  the 
directions  to  the  best  vertex  from  the  other  vertices. 
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Proposition  3.11.  There  exists  c  >  0  such  that  given  any  rj  >  0^  we  can  find 
1/  >  0  such  that  if  ||  1|  >  ^  Ak  <  i',  then 


(^k 


-9{'Vkf(^k  >  c  I  g{v^) 

Proof.  Recall  that  the  reflection  step  has  the  form 

=  EpI  {vt  -  4) ,  pi  >0,P^>  0, 


2  =  1 


SO 


(15) 


-s{vkf(^k  = 

2=1 

=  -Pk9{4kf  («fc  -  ^it)  -  E  49{4kf  -  4) 

i=\ 

.  find  ui  >  0  such  that  if  Ak  <  I'l,  then 


Using  Proposition  3.10  we  can 
(16)  -s(vinvt-vi)>C  g(vt) 

li  II  II 

),  Proposition  3.9  allows  us  to  find  V2  so  that  if  Ak  <  1^2, 


4-4 


Meanwhile,  given  any  £  >  0, 
then 

9i4f{4  -vi)<e  I  ff(4)  1 1 4  -  4  „ 

for  all  i  =  1, •  •  •, n  —  1.  We  will  choose  s  felicitously  in  a  moment.  In  the  meantime, 
applying  (11)  to  the  previous  bound  yields 


p(4) 


4-4 


(17)  9(44(4  —  4)  ^  II  II  II  ''k  "k  II  • 

Now  let  t'  =  mm(i'i,i'2)  and  suppose  <  u.  Returning  to  (15),  we  can  apply 
(16)  and  (17)  to  obtain  the  bound 

-9(44 (^k  >  Cpi  9(4)  1 4  -  4 1  -  E  4n*  II  ^(4)  I  4  -  4 

2=1 

>  -  £7*  E  4-)  II  s(4)  II II 4  -  4 

Now  we  choose  £  >  0.  Let 


and 


=  minp^ 
k 


Cl.  p* 

£  <  ■:r  —  nun  ^ — r. 

2  7*  Li=i  Pk 


We  are  assured  of  a  nonzero  p*  and  £  because  Pfc  =  [5fc  Rk]  G  M,  and  M  is  a  finite 
set  of  matrices,  so  the  minimum  in  each  of  the  two  preceding  relations  is  taken  only 
over  a  finite  set.  With  this  choice  of  £  and  p*,  we  obtain 


(18) 


-  9(41^ (^k  >  jP* 


9(4)  II II 4  -  4 
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Finally,  we  must  relate  the  norm  of  to  that  of  <Tk.  We  have 

=  -  4) 

t=l 

with  p\  >  0,  so 

n 

(19)  \\<7k\\<J2p^k\\4-4\ 

i=l 

Now,  the  vectors  ^  ^  ^ire  the  columns  of  A/-55fc,  possibly  permuted: 

[(»” --i)  ■•■«-»")] 

where  Ha;  is  a  permutation  matrix.  Consider  any  invertible  n  x  n  matrix  A  with 
columns  oi, •••,«„.  For  any  i,j,  from  Cj  =  A'^aj  and  ai  =  Aci  we  obtain  the 
inequalities  1  <  ||  A~'^  ||  ||  aj  ||  and  ||  ai  ||  <  ||  A  |1,  whence  ||  a,-  ||  <  «;(A)  ||  Oj  ||.  Thus 
for  any  i  we  have 

I -  4  I  <  K(AA55fcnA)  II 4  -  4  I  =  <BSk)  II 4  -  4  I  • 

The  latter  equality  holds  because  11  a  is  an  orthogonal  transformation.  Returning  to 

(19) ,  we  then  have 

Iktiis  + 

\  i=l 

Since  there  are  only  finitely  many  choices  for  Sk  a-nd  the  we  can  find  a  constant 
/i  ,  independent  of  fc,  such  that 

(20)  ||aA.||<ii:||4-4|  • 

The  result  then  follows  from  (18)  and  (20).  □ 

A  consequence  of  the  preceding  proposition  is  that  if  4  is  not  a  stationary  point 
and  if  Ak  is  sufficiently  small,  then  the  reflection  step  wiU  improve  upon  the  best 
objective  value  f{vl),  and  not  just  f{xk).  However,  we  have  chosen  to  pose  Proposition 
3.3  in  terms  of  Xfc,  which  is  not  necessarily  4-  Proposition  3.3  now  follows  as  a  corollary 
of  Proposition  3.11  and  the  uniform  continuity  of  g: 

Corollary  3.12.  There  exists  c  >  0  such  that  given  any  rj  >  0,  we  can  find 
1/  >  0  such  that  if  ||  ^(4)  II  >  ^ 

-g{xkf<yk  >c\\g{xk)  II  11  ak  II . 

Proof.  Choose  >  0  so  small  that  if  <  I'l,  then  the  conclusion  of  Proposi¬ 
tion  3.11  holds.  Then 

T 

-g{xkf(Tk  =  -g{vk)'^ak+ (g{vt)-g{xk))  ak- 

>  C  II  5(4)  II II  II  -  II  5(4)  -  gi-^k)  II II  cTfc  II . 
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Next  choose  r/2  >  0  so  small  that  if  <  2^2?  then 


and 


9{vk)  -  9{^k) 


9(4)  >  2  II  II  • 


Then,  if  A*,  <  mm{i/i,V2), 

-g{xkf<7k  >  j  II  9(^k)  II II  CTk  II  • 


□ 

3.2.  Finding  an  acceptable  step.  From  this  point  on  the  convergence  analyses 
for  positive  basis  pattern  search  methods  and  rank  ordered  pattern  search  methods 
are  identical  and  follow  more  or  less  directly  from  results  developed  in  [7]  and  [12]. 

The  following  two  results  come  from  [12],  to  which  we  refer  the  reader  for  the 
proofs.  The  first  result  indicates  one  sense  in  which  Ak  regulates  step  length. 

Lemma  3.13  (Lemma  3.1  from  [12]).  There  exists  a  constant  C*  >  0,  indepen¬ 
dent  of  k,  such  that  for  any  trial  step  7^  0  produced  by  a  pattern  search  method  for 
unconstrained  minimization  (Algorithm  1)  we  have  ||  $1  ||  > 

We  also  recall 

Lemma  3.14  (Lemma  3.6  from  [12]).  If  there  exists  a  constant  C  >  0  such 
that  for  all  k,  C  >  ||c),||,  for  all  i  =  1,  •••,?,  then  there  exists  a  constant  'll;*  >  0, 
independent  of  k,  such  that  for  any  trial  step  s\  produced  by  a  pattern  search  method 
for  unconstrained  minimization  (Algorithm  1)  we  have  Ak  >  t^*||5fc||- 

We  are  now  ready  to  state  and  prove  the  main  result  for  this  section. 

Proposition  3.15.  Suppose  that  L{xo)  is  compact  and  that  f  is  continuously 
differentiable  on  an  open  neighborhood  fi  of  L{xq).  Then  given  any  J?  >  0,  there  exists 
^  >  0,  independent  of  k,  such  that  if  Ak  <  S  and  ||  g{xk)  1|  >  Vt  then  either  the 
positive  basis  or  rank  ordered  pattern  search  method  will  find  an  acceptable  step  Sk; 
i.e.,  f{xk  +  Sk)  <  f{xk)- 

If,  in  addition,  the  columns  of  the  generating  matrices  remain  bounded  in  norm 
and  we  enforce  the  Strong  Hypotheses  on  Exploratory  Moves,  then,  given  any  7/  >  0, 
there  exist  ^  >  0  and  c  >  0,  independent  of  k,  such  that  if  Ak  <  b  and  ||  g{xk)  ||  >  t], 
then 


/(a^i+i)  <  f{xk)  -  c  II  g{xk)  II  II  sfc  II . 

Proof  Proposition  3.3  assures  us  that  we  can  find  >  0  such  that  if  Afc  <  ^i, 
then  there  is  a  step  4  ^  I^kBTk  for  which 


(21) 


g{xk)^s\  <  -c  II  g{xk) 


Meanwhile,  Proposition  3.5  says  that  we  can  choose  ^2  >  0  such  that  if  A^  <  ^2?  then 


(22) 


f{xk  +  4)  -  fi^k)  <  g{xk)'^si  +  ^Tj 


Sk 
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Thus,  if  Afc  <  (5  =  min(5i,^2),  (21)  and  (22)  yield 

f{Xk  +  Sk)-  f{xk)<-^\\g{xk)\\\  Sk  , 

and  so  f{xi)  =  f(xk  +  s{)  <  f{xk)  for  at  least  one  s].  G  AkBTk.  The  Hypotheses  on 
Exploratory  Moves  guarantee  that  if 

min  {  f(xk  +  y)  I  y  £  AkBTk  }  <  f{xk), 

then  f{xk  +  Sk)  <  fixk)-  This  proves  the  first  part  of  the  Proposition. 

If,  in  addition,  we  enforce  the  Strong  Hypotheses  on  Exploratory  Moves,  then  we 
actually  have 

/(xjt  +  -Sfc)-/(a:fc)  < II  4  • 

Lemma  3.13  then  ensures  that 

f{xk  +  Sk)  <  f{xk)  —  II  gi^k)  II  • 

Applying  Lemma  3.14,  we  arrive  at 

f{xk  +  Sk)  <  fixk)  -  |c*V’*  II  gi^k)  II II  Sk  II , 

which  is  the  desired  estimate.  0 

The  following  coroUary,  which  follows  from  Proposition  3.15  and  the  update  for 
Ak,  corresponds  to  Proposition  3.4  in  [12]: 

Corollary  3.16.  Suppose  that  L{xo)  is  compact  and  that  f  is  continuously 
differentiable  on  an  open  neighborhood  Q  of  L{xo).  //hminffc_»+oo  ||  gixk)  ||  0, 

then  there  exists  a  constant  A*  >  0  such  that  for  all  k,  Ajt  >  A* . 

If  liminffc_+co  ||  gixk)  ||  7^  0,  then  Proposition  3.15  says  that  there  is  a  uniform 
bound  ^  >  0  such  that  once  Ajt  <  6,  the  pattern  search  algorithm  wiU  necessarily  find 
an  acceptable  step.  Since  we  reduce  Ak  only  if  we  have  an  unsuccessful  iteration,  this 
means  we  would  at  some  point  stop  reducing  Afc. 

3.3.  The  algebraic  nature  of  the  iterates.  The  iterates  produced  by  pattern 
search  methods  have  a  specific  algebraic  form.  The  rank  ordered  and  positive  basis 
pattern  search  methods  inherit  this  basic  algebraic  structure.  The  next  result,  a  proof 
of  which  can  be  found  in  [12],  is  key  to  the  convergence  of  pattern  search  methods. 

Theorem  3.17  (Theorem  3.2  from  [12]).  Any  iterate  x^  produced  by  a  pattern 
search  method  for  unconstrained  minimization  (Algorithm  1)  can  be  expressed  in  the 
following  form: 

N-l 

x;,-  =  xo+  ^k, 

k=0 

where 

•  xq  is  the  initial  guess, 

•  /3/a  =  T,  with  a,/?  G  N  and  relatively  prime,  and  r  is  as  defined  in  the 
algorithm  for  updating  Ak  (Algorithm  2), 

•  rLB  ruB  depend  on  N , 
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•  Ao  is  the  initial  choice  for  the  step  length  control  parameter, 

•  B  is  the  basis  matrix,  and 

•  Zk  ^  Z",  A;  =  0,  •  •  • ,  iV  —  1. 

The  next  theorem  combines  the  strict  algebraic  structure  of  the  iterates,  the  simple 
decrease  condition,  and  the  algorithm  for  updating  Afc,  to  reach  a  conclusion  about 
the  limiting  behavior  of  Afc. 

Theorem  3.18.  Suppose  that  L{xq)  is  compact.  Then  liminffc_v+oo  A*  =  0. 

The  proof  is  identical  to  that  of  Theorem  3.3  in  [12].  Briefly,  Theorem  3.17  says  that 
the  iterates  lie  on  a  rational  lattice.  If  liminf^;-^+oo  A*  ^  0,  then  there  can  be  only  a 
finite  number  of  distinct  points  visited  by  the  algorithm.  However,  Proposition  3.15 
says  that  unless  we  are  at  a  stationary  point,  we  will  eventually  choose  a  new  iterate 
with  a  strictly  lower  objective  value;  this  is  at  odds  with  the  fact  that  we  only  visit  a 
finite  number  of  distinct  points. 

3.4.  The  proofs  of  Theorem  3.1  and  Theorem  3.2.  The  conclusion  of  the 
proof  of  Theorem  3.1  is  identical  to  that  of  Theorem  3.5  in  [12].  Suppose  that 
liminffc_*+oo  1|  g{xk)  ||  #  0.  Then  Corollary  3.16  tells  us  that  there  exists  A,  >  0 
such  that  for  aU  k,  Ak  >  A*.  But  this  contradicts  Theorem  3.18. 

The  proof  of  Theorem  3.2  follows  that  of  Theorem  3.7  in  [12],  to  which  we  refer 
the  reader. 

4.  Concluding  remarks.  We  believe  that  versions  of  the  two  classes  of  algo¬ 
rithms  we  have  introduced  in  this  paper  can  be  developed  for  bound  constrained 
minimization  as  well.  The  positive  basis  pattern  search  methods  for  unconstrained 
minimization  require  a  positive  basis  for  R”;  the  correct  analog  for  bound  constrained 
minimization  should  require  a  positive  basis  for  the  tangent  cone  of  the  feasible  region 
at  each  iterate.  The  work  in  [7]  uses  such  a  basis  that  appears  to  be  maximal  in 
size.  The  theory  of  positive  linear  dependence  used  in  the  unconstrained  case  suggests 
a  line  of  development  to  sharpen  the  results  in  [7],  which  may  in  turn  reduce  the 
computational  cost  per  iteration  in  the  bound  constrained  case. 

Similarly,  the  heuristic  of  approximating  the  direction  of  steepest  descent  using 
the  best  and  worst  of  n  -f  1  vertices  (determined  by  their  objective  values)  should  also 
be  applicable  to  the  bound  constrained  case.  There  we  would  only  consider  feasible 
points  and  develop  a  crude  approximation  for  the  direction  considered  in  gradient 
projection  algorithms. 
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